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Abstract 

This discussion constructs a survey data quality strategy 
for institutional researchers in higher education in light of total 
survey error theory. It starts with describing the characteristics 
of institutional research and identifying the gaps in literature 
regarding survey data quality issues in institutional research 
and then introduces the quality perspective of a survey process 
and the major components of total survey error. A proposed 
strategy for inspecting survey data quality is presented on 
the basis of five types of survey error and the characteristics 
of typical institutional research survey projects. The strategy 
consists of identifying quality measures for each type of survey 
error, and then identifying quality control and quality assurance 
procedures for each of the quality measures. The discussion 
concludes with the implications of the strategy for institutional 
researchers and some closing thoughts. A checklist for 
inspecting survey data quality is provided in the appendix. 


Since the 1 960s, institutional research (IR) has emerged 
as a profession and gradually become an organizational 
function of higher education institutions in many parts of 
the world. Its presence in higher education is a response to 
changing demands of society for its institutions, such as calls 
for increased accountability and efficiency (McLaughlin & 
Howard, 2004). Initially, IR appeared as a decentralized set of 
activities conducted in various offices throughout university/ 
college campuses; however, its function frequently has become 
centralized in institutional research offices. 

Institutional research helps answer three questions essential 
to the sustained development of an organization: Where is 
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the organization at this moment? Where is the 
organization going? And how can the organization 
best arrive at its desired end? (Middaugh,Trusheim, 
& Bauer, 1994, p.1). Institutional research fosters 
organizational learning (Leimer, 2009), and IR has 
increasingly become a core administrative function 
through its integration into the strategic planning 
and assessment processes of the institution (Morest, 
2009). It is anticipated that institutional researchers 
will not only continue to fulfill their core function 
by converting data into information but will also 
become change agents by actively engaging in 
the process of managing and leading institutional 
change (Swing, 2009). 

What is institutional research about? In his 
seminal monograph on institutional research, Saupe 
(1990) defines institutional research as "research 
conducted within an institution of higher education 
to provide information which supports institutional 
planning, policy formation and decision making" 

(p. 1 ). As such, the main product of institutional 
research is information. According to Middaugh et 
al. (1 994), there are multiple institutional research 
measures including those related to organizational 
inputs (e.g., students, faculty and staff, facilities, 
and revenues), processes (e.g., academic programs, 
program completion, quality, productivity, and 
strategic planning), outputs (e.g., graduates and 
student outcomes) and the external environment 
(e.g., financial considerations, employment market, 
government concerns, and regional accreditation). 
The task of institutional researchers involves 
converting complex data to actionable information 
(Anderson, Milner, & Foley, 2008) and concerns 
three interdependent forms of "organizational 
intelligence" (Fincher, 1 985): technical/analytical 
intelligence, issues intelligence, and contextual 
intelligence (Terenzini, 1993). Institutional research 
draws from various methodologies including 
applied research, program evaluation, policy 
analysis, and action research, depending on 
questions it needs to address (Saupe, 1 990). 

Institutional research distinguishes itself from 
higher education research in that institutional 
researchers are more concerned about knowledge 
about a specific institution or system of institutions 


whereas higher education researchers are more 
interested in advancement of theory and practice in 
higher education in general (Saupe, 1990; Terenzini, 
1 993). Another difference is the importance of 
the utilization or application of the information 
produced through institutional research. The 
use of information is viewed as an essential 
component in the life cycle of an institutional 
research activity. Along with other stages of design, 
collection, preparation, analysis, and dissemination, 
application of the research results and feedback 
received are the "primary determinants] of success" 
(Borden, Massa, & Milam, 2001, p. 200). 

Institutional research involves three data 
sources: institutional information systems or 
administrative data (e.g., student enrollment data 
or faculty data), external data sources (regional 
or national data, e.g.. Integrated Postsecondary 
Education Data System in the United States), and 
data collected as responses to various surveys and 
queries (Borden et al., 2001). Frequently, the survey 
used will be a locally developed one. Although a 
locally prepared survey may be the best option for 
obtaining the needed information, information 
users in higher education institutions generally 
have higher levels of trust in research derived 
from administrative data than that derived from 
survey data. A similar pattern is also found among 
higher education researchers and policy makers, 
who traditionally have less confidence in softer 
and more subjective measures such as survey data 
based on perceptions (Gonyea, 2005). At the same 
time, more than ever, the demands for external 
assessment and internal assessment have fueled 
an increase in the demand for quality survey data 
(Porter, 2004). Further, the validity of commonly 
used college student surveys has been called into 
question and more rigorous and diligent efforts in 
survey design and evaluation are in demand (Porter, 
2011). To increase the acceptance and use of survey 
data, further efforts are required to improve survey 
data quality in both higher education research and 
institutional research. 

Unfortunately, literature that examines survey 
data quality issues in a holistic way for the purposes 
of institutional research professionals is limited. 
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There seem to be two gaps in the existing efforts 
to improve on survey methodology. First, while 
there is a large amount of literature addressing 
various aspects of a survey project (Croninger & 
Douglas, 2005; Gonyea, 2005; Porter, 2004; Presser 
et al., 2004; Sanchez, 1992; Thomas, Heck, & Bauer, 
2005), there is lack of synthesis of this literature 
into a conceptual model for the purposes of quality 
control. Because of this lack of synthesis, issues of 
survey methodology have yet to be approached 
from a survey quality perspective. Because survey 
quality is a complex issue with multiple dimensions, 
a lack of synthesis may be understandable without 
a conceptual model of survey quality. When one 
particular area is addressed in depth, it is hard 
to cover other areas and cover the breadth of 
the topic. However, this synthesis of knowledge 
is necessary for any kind of survey project if the 
researcher is serious about the survey data quality, 
regardless of whether the research concerns multi- 
institutional surveys such as the National Survey of 
Student Engagement (NSSE) or in-house surveys 
specific to one institution. 

The second gap, and a related issue, is the lack 
of a comprehensive conceptual model of survey 
errors in the field of institutional research. While 
"serious" institutional researchers are making 
efforts to improve survey data quality, they often 
do not rely on any theories on survey methodology 
(Gonyea, Korkmaz, BrckaLorenz, & Miller, 2010). A 
survey methodology theory — total survey error, to 
be described later in this paper — is a very helpful 
way to address survey data quality issues. However, 
not much can be found in the current institutional 
research literature in reference to total survey error 
theory and its applications in institutional research. 
Although in a few publications, survey issues in the 
field of institutional research have been approached 
from the point of view of survey errors (Umbach, 
2004, 2005), discourse of data quality is limited. 

As such, it is meaningful to look at survey issues 
in institutional research from the perspective of 
data quality through the lens of a total survey error 
theory as it provides a theoretical framework for 
optimizing surveys (Biemer, 201 0) and the central 
organizing structure for survey methodology 


(Groves & Lyberg, 2010). Such a view informs the 
following discussion, which aims to incorporate the 
total survey error paradigm into the survey data 
quality issues in institutional research. 

This task is more pressing given the unbalanced 
efforts among institutional researchers to improve 
the quality of data with which they are working. 
How institutional research provides information 
support to a higher education institution is 
illustrated in a book that carries much weight in the 
field of institutional research. People, Processes, and 
Managing Data (McLaughlin & Howard, 2004). The 
book presents an information support cycle that 
involves three stakeholders through five stages of 
information management: the custodian/supplier 
(focusing on integrity of the data), the broker/ 
producer (transforming the data into information), 
and the manager/user (taking the information 
and applying it to the situation); the center of 
this information support cycle is quality decision- 
making (Figure 1).The institutional research 
function is identified as being mostly aligned to 
the information broker role while relating to the 
data custodian and the data user (p. 1 7). The book, 
however, only speaks to one of the data sources 
used for institutional research — administrative data 
in an institution — and does not address data that 
are generated from survey research projects. So, the 
question remains unclear as to how the institutional 
research function fits into the information support 
cycle when survey data are involved. 



Figure 7. Information support cycle in institutional 
research. 

(Source: McLaughlin & Howard, 2004) 
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In this context, this following discussion draws 
upon the total survey error theory and presents 
a survey data quality strategy for the purposes of 
institutional research. In the following section, 

I briefly introduce the quality perspective of a 
survey process and the major components of 
total survey error. Then, I present the proposed 
strategy for survey data quality in a summary 
table with explanations to follow, and I end with 
discussions about the implications of the strategy 
for institutional researchers. It should be noted 
that the intention is to focus on the big picture 
and the breadth of survey quality data issues in 
institutional research, rather than the depth of each 
issue. Readers interested in a particular topic are 
encouraged to consult articles addressing individual 
issues for details. 

The Quality Perspective of a Survey 
Process and the Total Survey Error 

Doing a survey can be viewed in two ways. 

First, it can be viewed from a process perspective, 
wherein one might examine all the steps and 
decisions that are required in a survey project, 
including determining research objectives based on 
information needs, determining sampling methods, 
developing the survey instrument, administering 
the survey, conducting data analysis, and finally 
producing the survey report (Alreck & Settle, 1 985; 
Biemer & Lyberg, 2003). This approach describes 
the survey process as a procedure that is mostly 
sequential but with iterations. 

The second approach to a survey process is to 
view it from a quality perspective. This approach 
does not focus on how best to implement each 
step in a survey process but concentrates on 
what problems may occur in each step and how 
to overcome or minimize the occurrence of those 
problems (Groves et al., 2009). In other words, it 
intends to examine errors that may occur in the 
survey process with a view of minimizing those 
errors, thereby improving survey data quality. As 


such, the quality perspective is linked with survey 
errors, and this is where the concept of total survey 
error comes in. The goal of an optimal survey design 
is simply stated as "minimizing the total survey error 
subject to costs and timeliness constraints" (Biemer, 

2010, p. 821). 

How the quality perspective of a survey process 
compares with the process perspective is shown 
in Figure 2. The quality perspective of a survey 
process consists of a measurement track and a 
representation track (Groves et al., 2009), and both 
tracks move from the abstract to the concrete. The 
measurement track starts with the construct, and the 
representation track begins with target population. 

As a survey project proceeds from information needs 
to data processing and report generation from the 
survey process point of view (Alreck & Settle, 1 985), 
the two tracks move down and converge at the 
point of obtaining survey statistics. 

Total survey error is "the difference between 
a population mean, total or other population 
parameter, and the estimate of the parameter based 
on the sample survey (or census)" (Biemer & Lyberg, 
2003, p. 36). Survey errors can be organized in 
different ways. Some categorize them into sampling 
error and nonsampling error (Biemer & Lyberg, 
2003); others classify them into observational 
error associated with the measurement process 
(starting with a construct and ending with an edited 
response) and nonobservational error associated 
with the representation process (starting with 
target population and ending with postsurvey 
adjustments; Groves et al., 2009). Regardless how 
survey errors are organized (Biemer & Lyberg, 2003; 
Groves et al., 2009; Weisberg, 2005), the total survey 
error consists of the following five types of survey 
error: measurement error, coverage error, sampling 
error, nonresponse error, and postsurvey (i.e., data 
processing or adjustment) error. These types of 
errors are shown in Figure 2 and are described in 
more detail later. 

Each type of survey error creates the risk of 
variable errors and systematic errors 1 , which can 


1 The value of a question item for a particular person is either higher or lower than the true value of the person; the cumulative 
effects of all errors are positive for some respondents and negative for others. When the variable negative errors in the sample 
observations tend to cancel the positive errors, these errors are called variable errors. Errors that do not sum to zero when the 
sample observations are averaged are referred to as systematic errors (Biemer & Lyberg, 2003, pp. 46-47). 
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(Source: Alreck & Settle, 1985, p. 31) 
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(Source: Adapted from Groves, et al., 2009, p. 48) 


Figure 2. Comparison between the Process Perspective and the Quality Perspective of a Survey Process 


Table 1 

Risk of Variable and Systematic Errors by Major Error Source in the Institutional Research Context 


Types of Survey Error 

Risk of Variable Error 

Risk of Systematic Error 

Measurement error 

High 

High 

Coverage error 

Low 

Low 

Sampling error 

High 

Low 

Non-response error 

Low 

High 

Post-survey/Data processing error 

High 

Low 


Note. Modified from Introduction to Survey Quality (p. 59), by P. P. Biemer and L. E. Lyberg, 2003, Hoboken, NJ: John Wiley & Sons, Inc. 
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result in error variance and bias 2 respectively. Error 
variance and bias are two measures of data quality, 
with variance being easier to measure and control 
than bias (Bailar, Herriot, & Passel, 1 982; Czaja & 

Blair, 2005). Table 1 identifies the risks of variable 
error and systematic error for the five types of 
survey errors. 

In general, among the five types of survey 
error, the sampling error is subject to a low risk 
of systematic errors whereas the other four types 
likely have a high risk of systematic errors (Biemer 
& Lyberg, 2003, p. 59). This is because in a typical 
survey using reasonable sampling methods, the risk 
of systematic error from the sampling error is quite 
small, while the risk of variable error is inevitable. 

In contrast, the risk of systematic error from the 
measurement error is high as a questionnaire may 
be poorly designed. The risk of systematic error 
from the coverage error is high as a group of people 
may be completely missing from the sampling 
frame. 

In a typical institutional research survey project, 
the target population is generally either students 
or faculty/staff members of the institution, which 
is usually a finite population. The sampling frame 
is usually available from the student information 
system or the human resources database, and this 
advantage has reduced the risk of the systematic 
error from coverage error. The target population 
usually uses emails and online resources regularly. 
This has made it easier to administer web surveys 
to university/college students or faculty/staff 
than the population outside this context, and has 
made web surveys a commonly used survey mode 
within higher education institutions. Furthermore, 
higher education institutions usually have access 
to relatively advanced data entry and processing 
resources (e.g., software and research expertise), 
which may help reduce data processing errors. 
Given these characteristics, the risk of variable 
errors and the risk of systematic errors in each type 
of survey error may be different from those in a 
survey project in other contexts. Table 1 shows an 


indication of the risk of variable and systematic 
errors in the five types of survey error in a typical 
institutional research survey project. 

Survey Data Quality Strategy 

The survey data quality strategy proposed in 
this paper addresses the five identified types of 
survey error in a framework of quality assurance and 
quality control in surveys (Lyberg & Biemer, 2008). 

There is a fine distinction between the two 
concepts: "Quality assurance ensures that processes 
are capable of delivering good products, while 
quality control ensures that the product actually 
is good" (Lyberg & Biemer, 2008, p. 426). As such, 
quality assurance is related to the survey process 
while quality control is associated with the survey 
product. Therefore, survey researchers are pursuing 
both process quality and product quality. Survey 
quality is assured by using dependable processes 
(process quality), and these processes lead to 
good product characteristics (product quality) 
(Biemer & Lyberg, 2003). In light of this framework, 
the characteristics of quality survey data that 
have been collected (i.e., the product) need to be 
defined for the purposes of quality control, and the 
characteristics of a quality survey process should be 
identified for the purpose of quality assurance. 

With these considerations in mind, three 
components comprise the proposed survey data 
quality strategy: quality measures, which are the 
characteristics or indicators of quality survey data; 
quality control procedures, which are used to inspect 
the various aspects of the survey and the data and 
to examine whether the aspects and obtained data 
have those characteristics of quality, as specified 
by the quality measures; and quality assurance 
procedures, which are used to inspect the survey 
process and check whether certain procedures were 
implemented during the survey process to ensure 
that the resulting survey data set will have those 
characteristics of quality survey data, as specified by 
the quality measures. 


2 Variance means the differences manifested in repeated trials of a procedure, and reflects the degree of spread or variation in a 
sample. Bias is an estimate that is consistently higher or lower than the true population value (Czaja & Blair, 2005). 
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In Table 2, the survey data quality strategy 
is presented in a matrix with the five types of 
survey error as rows, and the three areas of quality 
inspection as columns. The strategy is also aligned 
with the two approaches to addressing the issue 
of survey errors: error measurement and error 
reduction (Czaja & Blair, 2005). The quality control 
procedures are aimed to measure and assess survey 
errors, and the quality assurance procedures are 
intended to reduce survey errors. 

The following sections identify the quality 
measures for each of the five types of survey error 
and elaborate on the corresponding quality control 
and quality assurance procedures in the context of 
institutional research. Table 2 serves as a summary 
of these measures and their related procedures. 

Quality Inspection for Measurement Error 

Measurement error occurs when there are 
differences between the responses obtained and 
what was to be measured. In reference to Figure 2, 
measurement error represents the gap between 
the construct and the measurement, and the gap 
between the measurement and the response; these 
gaps may occur in the processes of instrumentation 
and data collection. Relating to the measurement 
error, quality survey data are characterized by three 
indicators: reasonably good validity, reasonably 
good reliability, and minimized response bias. 

Quality control procedures. As the sine qua 
non of measurement, validity is the extent to 
which the survey measure actually reflects the 
intended construct. 3 Assessing validity is mainly 
considered to inspect correlations. 4 As construct 
validity subsumes content and criterion-related 
validity (Borden & Young, 2008), it has become an 
overarching concept when thinking about validity. 
When the measurement is consistent with the 
theoretical concept behind it, then the data 


have construct validity. Construct validity has two 
measures: convergent validity (measured by a 
positive correlation between the survey responses 
with responses to similar questions in other surveys) 
and discriminant validity (measured by a relatively 
low correlation between the survey responses and 
answers to other questions that measure a different 
construct). While the multitrait-multimethod matrix 
has been recommended to assess the convergent 
and discriminant validity (Campbell & Fiske, 1 959), 
actually implementing it may be too monumental 
a task for institutional researchers. However, its 
implications on performing triangulation to assess 
the construct of interest still apply. This means that 
institutional researchers should inspect question 
items using different observation techniques in 
order to verify that the question items actually 
measure the same concept or the construct. Factor 
analysis is a statistical technique that is helpful for 
inspecting construct validity (Comrey & Lee, 1 992; 
Gregory, 2004). 

Reliability is "a measurement of the variability 
of answers over repeated conceptual trials" (Groves 
et al., 2009, p. 281). It addresses the question of 
whether respondents are consistent or stable in 
their answers, and it is therefore also known as 
response variance (Groves et al., 2009). Reliability is 
often computed as the correlation of two survey 
estimates, and reliability in the responses can 
be assessed in three ways: internal consistency 
(typically measured by Cronbach's alpha), split-half 
reliability, and test-retest reliability (both typically 
computed by Spearman-Brown coefficient). For 
qualitative responses, there is also inter-rater 
reliability to assess the reliability of the scoring. 

Response bias is a systematic deviation or 
discrepancy between the sample estimate and the 
true population parameter; in other words, 


3 1 acknowledge the myriad types of validity and also recognize the evolving concept of validity (Borden & Young, 2008). As my 
discussion in this section is concerned with measurement issues in survey data, I focus on validity only as the sine qua non of 
measurement. 

4 Other approaches to validity may be used to examine validity issues. For example, Porter (2011) has used an argument-approach 
to validity. 


Table 2 

Summary of the Major Components in the Survey Data Quality Strategy 
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the respondent mean response is consistently 
higher or lower than the true mean score of the 
population. Response bias can come from sources 
such as mood, social desirability, language difficulty, 
extreme-response sets, and acquiescence (Crano & 
Brewer, 2002, pp. 54-55). 

There are two ways to assess response bias. One 
is to compare survey data with data or information 
from sources external to the survey. An example of 
this method is to check with the stakeholder or the 
sponsor of the survey project to find out whether 
the survey findings about a particular question are 
far different from their experience or knowledge. 
Another way of assessment is to evaluate the 
occurrence of certain response tendencies, such 
as respondents giving socially acceptable answers, 
avoiding using the extreme response categories 
of a rating scale, or giving the same answer to 
all alternatives in a rating scale (known as strong 
satisficing; Groves et al., 2009). 

Quality assurance procedures. Measurement 
error can be reduced through the following 
techniques. First, researchers should construct a 
good survey that is based on a well-supported 
theory or conceptual framework, and work to have 
high quality question wording and questionnaire 
structure. As an institutional research survey project 
is often initiated by certain institutional needs, the 
survey instrument tends to be constructed mainly 
from experience and less often from a literature- 
based conceptual framework. However, despite the 
applied nature of an institutional research project, a 
literature review should be part of the survey design 
process. 

Second, cognitive interviews should be 
conducted to ensure the target population 
understands the questions in the same way as the 
questionnaire was intended. 

Third, adequate respondent behavior involves 
optimal completion of the cognitive process 
and sufficient motivation on the part of the 
respondent; therefore, the questionnaire should 
be designed and administered with a view to 
ensuring participants actually go through the 
four components of the mental processing in 
answering survey questions: comprehension, 


retrieval, judgment, and response (Tourangeau, 
Rips, & Rasinski, 2000). An example of failing to 
do so is strong satisficing, which may occur when 
respondents skip the retrieval and judgment steps 
and proceed to the response step (Groves et al., 
2009). Questions can be asked regarding how the 
respondents complete the questionnaire and how 
they self-evaluate their motivation and ability while 
answering the questionnaire. 

Fourth, for interviewer-administered surveys, 
procedures should ensure adequate interview 
behavior, which is measured by low interviewer 
variance and can be best analyzed with multilevel 
analysis (Loosveldt, Carton, & Billet, 2004). 

Quality Inspection for Coverage Error 

In reference to Figure 2, coverage error occurs 
when differences between the sampling frame and 
the target population exist. Quality survey data are 
characterized by minimized discrepancy between 
the sampling frame and the target population. 

Quality control procedures. Bias exists 
when some components in the target population 
are not available or accessible in the sampling 
frame. According to Groves et al. (2009), this may 
be caused by situations where elements in the 
target population do not, or cannot, appear in the 
sampling frame (i.e., undercoverage), units in the 
sampling frame are not in the target population 
(i.e., ineligible units), and several units in the frame 
are mapped onto the single element in the target 
population (i.e., duplication). Therefore, researchers 
should check for those cases. 

Another procedure for assessing the 
discrepancy between the sampling frame and the 
target population is to compare the specifications 
of the target population and the corresponding 
parameters of the sampling frame. As higher 
education institutions usually assign their students 
an institutional email address when they first enroll 
in their studies, the coverage error is not as big a 
threat to web surveys in institutional research as 
it is other fields in general (Couper, 2000). This is 
increasingly true as institutions use student email 
addresses in online course management systems. 
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Quality assurance procedures. Researchers 
need to develop a working definition and clear 
specifications of the target population and locate a 
readily available list that includes as many elements 
of target population as possible. In the context of 
institutional research, a typical target population 
is a student cohort with certain characteristics 
when they are applying for studies (perspective 
students), are currently enrolled in certain programs 
of study (current students), or have graduated 
within a certain time frame (graduates). A higher 
education institution usually has a well-established 
student database, so the sampling frame is often 
stable, complete, and accessible. Therefore, the 
risk of variable errors and systematic errors related 
to coverage error is generally low and more 
controllable. 

Quality Inspection for Sampling Error 

Sampling error is the difference between the 
survey estimate and the population parameter as 
a result of taking the sample instead of the entire 
population. In reference to Figure 2, sampling 
error represents the gap between the sampling 
frame and the sample. A good set of survey data 
is representative of the sampling frame by known 
demographic parameters. When probability 
sampling is used, margin of error is a commonly 
used measurement of the level of random sampling 
error. The commonly acceptable margin of error is 
less than 5% at the 95% confidence level. 

Quality control procedures. Sample 
representativeness can be determined by 
comparing the frequency distributions of the 
obtained sample and those of the sampling 
frame by certain demographic characteristics. If 
the difference in the frequency distributions is 
negligible, then the obtained sample is considered 
to represent the sampling frame by those indicators. 


Margin of error is affected by variance and 
sample size: the smaller the variance and the larger 
the sample size, the smaller the margin of error. 

For estimating the margin of error for the mean 
of an interval or ratio variable, we need to know 
the population standard deviation. * * * * 5 For a survey 
project in institutional research that usually has a 
finite population, the margin of error for a variable 
can be calculated on the basis of the total target 
population and the sample size 6 (Groves et al„ 

2009). 

Quality assurance procedures. The magnitude 
of sampling error is more controllable than other 
types of survey errors and, therefore, is considered 
as intentional error (Biemer & Lyberg, 2003). The 
sampling error can be controlled by making a good 
sample selection, which is featured by random 
selection of elements in the sampling frame and key 
subgroups of the population being well represented 
in the sample. Appropriate implementation of a 
sampling procedure requires four considerations: 
probability sampling, stratification, clustering, and 
sample size. Sampling bias can be easily removed 
by giving all elements an equal chance of selection; 
sampling variance is reduced when the sample 
size is big and the sample is stratified and is not 
clustered (Groves et al., 2009). 

An appropriate sampling strategy involves 
calculating a reasonable sample size. The sample 
size is determined by the sampling frame, desired 
margin of error, and anticipated response rate, 
analysis need for data breakdown, and the 
resources available for the survey. Table 3 shows 
an example to illustrate this. Please note that 
the margins of error presented in the table are 
calculated for a dichotomous variable using the 
maximum variance, that is, the standard deviation 
equaling 0.5 (Groves et al., 2009). 


5 The formula is: m = 1.96 * 2L ( a t the 95% confidence level) 

Vn 

(a: population standard deviation, usually obtained from literature reviews, expert knowledge, historical data, survey data or pilot 

studies; n=sample size). 

i s 

6 Margin of Error = 1 .96 * S = 1-96 * (y 1 - / * — ) (for a variable, at the 95% confidence level) 

yjn 

(S: Standard Error; -Jl- f ■ finite population correction factor, f=n/N; s: variable's standard deviation; n=sample size; N=population) 
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Table 3 

An Example of Determining the Sample Size for an IR Survey Project 


Margin of Error 
(95% confidence level) 

Sample Size 

Anticipated Response Rate 

Survey Invitees 

5.57% 

300 

20% 

1500 

4.80% 

400 

20% 

2000 

4.27% 

500 

20% 

2500 

3.88% 

600 

20% 

3000 

3.57% 

700 

20% 

3500 

3.32% 

800 

20% 

4000 

3.12% 

900 

20% 

4500 


Note. 1 . The sampling frame is 1 0,000 students. 

2. This table is produced with the assistance of an online sample size calculator tool (http://www.raosoft.com/samplesize. 

html) 


With a sampling frame of 1 0,000 students, if a 
researcher wants to make sure that the obtained 
margin of error is below the desired 5%, then the 
sample size needs to be 400 (margin of error of 
4.8%). If a response rate of 20% is anticipated, 
then a total of 2,000 students need to be invited 
to participate in the survey. If the 400 obtained 
sample is going to be broken down into subgroups 
for data analysis purposes, for example, for an 
institution with eight schools or faculties, then 
in each subgroup, there are 50 respondents. This 
size of respondents is acceptable for descriptive 
statistical analysis. However, if the researcher 
intends to conduct inferential statistical analysis 
or multivariate data analysis, s/he may want to 
consider increasing the sample size to 800 (the 
number depends on the questions to be answered), 
which results in a margin of error of 3.32% and 
a sample size of 4,000 when the anticipated 
response rate remains 20%. If the survey project 
is going to be administered in the web mode, the 
increased number of survey invitees will not matter 
because the bigger size will not add much to the 
total survey costs; however, when the mail survey 
is used, the increased costs from questionnaire 
distribution and data entry and processing will be a 
consideration in determining sample size. All these 


considerations will need to be balanced. Using an 
online calculation tool (e.g., http://www.raosoft. 
com/samplesize.html) can facilitate the process of 
computing the margin of error and the sample size 
when using dichotomous variables. 

Quality Inspection for Nonresponse Error 

In reference to Figure 2, nonresponse error 
represents the gap between the sample and 
the respondents. This error occurs when some 
people from the sample who were invited to 
participate in the survey do not respond to the 
survey request or fail to respond to some of the 
questions in the survey. Therefore, there are two 
types of nonresponse error: nonresponses at the 
unit level and nonresponses at the question item 
level (Biemer & Lyberg, 2003; Groves et al., 2009; 
Weisberg, 2005). Nonresponse bias occurs when the 
statistics computed from respondent data differ 
systematically from those based on the entire 
sample data in terms of known characteristics of 
interest. 

Quality control procedures for unit 
nonresponses. The measurement of nonresponse 
bias is the function of the nonresponse rate and 
the difference between the respondent and 
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nonrespondent estimates (Biemer & Lyberg, 

2003). As the nonresponse rate is the proportion 
of eligible survey recipients who did not respond 
to the survey, it can be calculated from the 
response rate. As such, a quality survey data set is 
characterized by a reasonable response rate and 
insignificant difference between respondents and 
nonrespondents with regard to the characteristics 
of interest to the survey. 

The difficulty in calculating the response 
rate usually lies in computing its denominator. 

For a survey project in institutional research, 
when the sampling frame is carefully extracted 
according to a well-defined target population, the 
response rate can be calculated in two primary 
ways: one is C/(C+NC+R+0), where C=completed 
questionnaires, NC=noncontacts, R=refusals, 
and 0=other nonrespondents (e.g., those who 
cannot understand due to language problems); 
the other is simply C/(S-NC), where C=completed 
questionnaires, S=sampled survey recipients, 
and NC=noncontacts. It can be seen from the 
two formulae that the sampled survey recipients 
actually consist of the survey respondents and 
three groups of nonrespondents (i.e., noncontacts, 
refusals, and other types of nonrespondents). 

Three methods can be used to assess 
the difference between respondents and 
nonrespondents. First, assess the degree to which 
nonrespondents will interact with the topics or 
issues of the survey. Usually, those who are highly 
involved with the topic of the survey are more 
likely to respond than those who are not, and 
those who have neutral opinions about the topic 
or have less experience are more likely to discard 
the questionnaire (Alreck & Settle, 1 985). For 
example, when an institution is conducting a survey 
targeted at its current students to find out how 
they have used its library services, it is important 
for the researcher to bear in mind that the obtained 
responses will over-represent the characteristics of 
those actual library users since those who have used 
the library services are more likely to respond to 
the survey. Therefore, it is erroneous to use the data 
to draw a conclusion about the library use pattern 
of all current students. Second, compare survey 


respondents' demographic characteristics with 
those of the sampling frame and find out whether 
the respondents under-represent some subgroups 
in the sampling frame and whether members of the 
under-represented groups tend to answer some 
of the substantive survey questions somewhat 
differently than others (Czaja & Blair, 2005). Third, 
examine the characteristics of the late respondents. 
Those who respond to the very last follow-ups 
may have similar characteristics to people who 
never respond, and hence, the responses from 
late respondents can be used to infer what the 
nonrespondents' would have answered (Suskie, 
1996). 

Quality control procedures for item 
nonresponses. Similar to unit nonresponse error, 
item nonresponse error is a function of the item 
nonresponse rate and the difference between 
item respondents and nonrespondents. Item 
nonresponse results in missing data. Flence, there 
are two characteristics of a quality survey data set 
at the question item level: a reasonable proportion 
of missing data for responses to each question 
item, and an insignificant difference between 
respondents and nonrespondents to each question 
item. 

Investigation into missing data and the 
difference between item respondents and 
nonrespondents involves item nonresponse 
analysis. A relatively large proportion of missing 
data for certain question items should be 
reported as a red flag for further investigation. 

Item nonresponse analysis involves examining 
(a) whether nonresponse occurrence is related 
to certain demographic characteristics of the 
respondents, or in other words, whether a particular 
group of respondents tends to fall short of 
substantial responses than others, and (b) whether 
the nonresponses to various question items are 
related. 

Item nonresponse analysis can be conducted in 
three ways: (a) calculate the proportion of missing 
data for each question; (b) decide the characteristics 
of missing data: missing completely at random, 
missing at random, or missing not at random 
(Allison, 2002); and (c) investigate variables with a 
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large proportion of missing data, and find out what 
may be the causes. 

Quality assurance procedures for 
nonresponses. Quality assurance procedures are 
derived from three types of unit nonresponse: 
noncontacts (failure to reach the survey recipients), 
refusals (recipients' decline of the survey request), 
and recipients' inability to participate (because 
of sickness, unavailability, language barriers, etc.; 
Biemer & Lyberg, 2003; Groves et al., 2009). The 
third situation also applies to item nonresponse; 
that is, some recipients may not be able to answer 
some of the questions because they find the 
questions difficult to understand, or they cannot 
retrieve requested information from memory, or 
the questions are beyond their capacity to answer. 
Survey nonresponse has been increasing, and much 
of the nonresponse is due to rising rates of refusals. 
A particular problem for institutional research 
survey projects is multiple surveys of students and 
the resulting survey fatigue (Porter, Whitcomb, & 
Weitzer, 2004). 

The reasons for nonresponses can be due to 
the social environment (e.g., survey fatigue), can be 
demographically related (e.g., male students may 
be less likely to respond to a survey request than 
females), and can be concerned with questionnaire 
design and survey administration. The factors 
related to survey design are more controllable than 
social or personal factors. 

Various techniques can be used to reduce 
the three types of nonresponse from survey 
design points of view (Czaja & Blair, 2005; Groves 
et al., 2009; Porter, 2004). Examples for reducing 
noncontacts are obtaining accurate contact 
information of the sampled recipients, and creating 
email messages that are not flagged by spam 
filters when using web surveys. Effective methods 
to combat refusals are notification about the 
survey prior to its launch, courteous initial contact 
messages (letter or email), the manner of requesting 
participation (e.g., the tone of the request, the 
signature, salience, and ensured confidentiality), a 
reasonable number of reminders, the appropriate 
timing of data collection, and the use of incentives. 
To help with the ability to participate, the survey 


instrument should be in reasonable length, be easy 
to read, and ask for relevant, available, or accessible 
information rather than inestimable information 
(Groves et al., 2009). A survey-friendly environment 
will also be helpful for increasing responses. Survey 
recipients are generally more likely to respond 
when a survey coordination mechanism is in place 
so that survey fatigue can be mitigated, and when it 
is known that survey results have been acted upon. 

Quality Inspection for Postsurvey Error 

Reliable findings and valid conclusions rely 
on correct data processing procedures for both 
individual data and aggregate data. Postsurvey 
error refers to all the errors that occur in the data 
processing process after the survey data are 
collected. In this data processing process, raw data 
are being transformed into information represented 
by survey statistics. As shown in Figure 2, the 
postsurvey errors occur when a response is turned 
into an edited response on the measurement track, 
and when adjustments are made to the respondents 
on the representation track, and when the edited 
or adjusted responses are converted into survey 
statistics. 

Quality control and assurance procedures. 

Various data processing activities after data 
collection can be organized into three types of 
procedure: data cleaning, data adjustment, and 
data analysis. Data cleaning involves inspecting 
accuracy of data entry, and checking for outliers 
and inconsistent data. Data adjustment involves 
considerations of using weights, handling missing 
data, and creating composite variables when 
needed. Data analysis involves conducting reliability 
and validity analysis, inspecting assumptions, 
choosing appropriate statistical techniques, and 
conducting statistical computing. When open- 
ended questions are used, coding is required, 
and involves inspecting weaknesses in coding 
structure and coder variance. It is usually hard 
to quantify how accurate and appropriate these 
procedures are while the researcher conducts 
data cleaning, data adjustment, and data analysis. 
Unlike the other types of survey errors, all these 
steps are in a researcher's control. Therefore, 
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whether each procedure is implemented correctly 
and appropriately relies, to a large extent, on the 
researcher's expertise and professional morale 
(for example, his/her diligence, rigorousness, 
meticulousness, and perseverance in data 
processing). Nevertheless, for quality control 
purposes, the procedures in data cleaning, 
adjustment, and analysis should be documented 
in detail to demonstrate evidence for quality in the 
data processing. 

Implications 

The survey data quality strategy presented 
in this paper has two practical implications for 
institutional researchers. 

First, survey data quality requires the use 
of multiple indicators. Survey data quality is 
multidimensional. This means that relying on 
one single indicator to evaluate survey data is 
a misleading practice, and the myth about the 
response rate is a good example. Institutional 
researchers sometimes hear such comments from 
information users as "With such a low response 
rate, the survey results are problematic" and "The 
response rate of the survey is high so the data 
represent the population well." 

The measures and procedures in the survey 
data quality strategy (see Table 2) suggest that a 
reasonably high response rate is simply one of the 
indicators for quality survey data though a very 
important one. To assess unit nonresponse error, 
the researcher also needs to consider whether 
there is any difference between responses and 
nonresponses in characteristics of interest. In 
evaluation of the quality of a survey data set, quality 
indicators derived from other types of survey error, 
such as any response bias that may affect the 
measurement error, and sample representativeness 
and margin of error to measure the sampling error, 
need to be included in addition to the response 
rate. Also, the response rate does not speak to the 
representativeness of survey responses, which is a 
separate indicator for nonresponse bias. Therefore, 
a relatively high response rate reduces the risk of 
nonresponse bias; however, this single indicator 
does not necessarily lead to the conclusion that 


the survey data have low nonresponse bias if the 
nonrespondents are very distinctive on the survey 
variable (Groves et al., 2009). As such, the strategy 
helps debunk some myths about survey data 
quality and encourages researchers to examine 
other quality indicators while focusing on one 
indicator such as the response rate. 

Second, it is important to document survey data 
quality. The survey data quality strategy lends 
more importance to quality documentation. The 
strategy proceeds from types of survey error based 
on the theory of total survey error; it consists of 
quality measures for each type of survey error and 
identifying quality control and quality assurance 
procedures that address each of the quality 
measures. Table 2 provides an organizational 
outline for deriving evidence of survey data quality 
and the procedures for collecting the evidence. 
Therefore, in the final analysis, the task of an 
institutional researcher is to collect evidence from 
the characteristics of the obtained survey data 
and the implemented survey process to convince 
information users that the survey data that have 
been collected are good for certain conclusions. 
The more evidence that is identified and presented, 
the greater the trust gained from the information 
users. This evidence collection process requires 
documentation. 

The information about the identified evidence 
relating to survey data quality is called metadata 
(i.e., data about data). Four types of metadata 
can be used to document survey data quality 
(Groves et al., 2009): definitional (investigated 
constructs, target population, sampling frame, and 
coding terminology), procedural (data collection 
procedures), operational (data cleaning, data 
adjustment, and data analysis procedures), and 
systems (data set format, file location, retrieval 
protocol, and codebook). 

In the report of an institutional research project, 
an appendix usually provides a detailed description 
of the methodology employed in the study (Bers 
& Seybert, 1 999). The appendix is a good place in 
a survey report to document evidence for survey 
data quality, and it should be an integral part 
of the report. A psychometric portfolio is also 
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recommended for reporting the evidence (Gonyea 
etal., 2010). 

The purpose of documentation is to 
communicate the characteristics of the survey 
data set and the procedures employed to obtain 
quality indicators, so as to build and enhance trust 
among information users (if there are any) in survey 
findings and help them interpret the findings 
in an appropriate way. Based on the elements 
in the survey data quality strategy in Table 2, 1 
have developed a checklist (see the Appendix) 
for institutional researchers to facilitate their 
documentation efforts. 

Concluding Thoughts 

This discussion has presented a survey data 
quality strategy in light of the theory of total survey 
error for the purpose of institutional research that 
is conducted in higher education institutions. 

The strategy consists of indicators of quality data 
(quality measures) and identifying procedures that 
are used to inspect the survey data and the survey 
process (that is, the quality control and quality 
assurance procedures) so as to measure and reduce 
errors. The strategy is summarized in Table 2, and 
the paper elaborates on each of its components; a 
checklist for institutional researchers is provided in 
the appendix. 

Here are two final thoughts about survey data 
quality issues. The first is related to how survey 
data quality fits in survey quality. As outlined by 
Lyberg and Biemer (2008, p. 428), survey quality 
is recognized as a three-level concept: product 
quality (i.e., "the set of product characteristics 
ideally established with the main users"), process 
quality (i.e., "a well-designed and tightly-controlled 
process"), and organizational quality (i.e., "reliable 
organizational characteristics to ensure that the 
organization is capable to develop dependable 
processes that can deliver quality products"). The 
three levels are interdependent (i.e., organizational 
quality is required for process quality, and process 
quality is required for product quality), and all the 
three levels contribute to quality decision-making. 


Survey data quality is actually part of the product 
quality of a survey and "is achieved through process 
quality" (Biemer & Lyberg, 2003, p. 24). The survey 
data quality strategy mainly addresses two of the 
three levels of survey quality: the product quality 
and the process quality. Organization quality 
is more concerned with organizational culture 
and information management, and involves 
information infrastructure for quality survey data. 

It is not addressed in the strategy presented in this 
discussion. 

The second closing thought relates to how 
survey data quality stands in McLaughlin's and 
Howard's (2004) information support cycle in 
institutional research (see Figure 1). When the 
survey data quality strategy proposed in this 
paper fits into the information support cycle, the 
institutional researcher actually takes the bulk of 
responsibilities in this cycle — the responsibilities of 
both the custodian and the broker, and implements 
the larger proportion of the information support 
cycle — from identifying concepts to delivering a 
report, while working with the manager. In contrast, 
when administrative data are used, the institutional 
researcher usually does not get as involved in the 
stages of collecting and storing data. Therefore, 
the role of the researcher is of greater importance 
in the information support cycle when a survey 
project is conducted. This may provide another 
reason for further investigating survey data quality 
issues and making greater efforts to improve survey 
data quality in order to better fulfill the information 
support function of institutional research. 

In the context where attention to detail 
and quality control is recognized as a personal 
and professional dimension of effectiveness in 
institutional research (Knight, 2010), it is hoped that 
this paper contributes to filling a gap in literature 
of survey quality control for institutional research 
purposes. 
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Appendix 

A Checklist for Inspecting Survey Data Quality in an IR Survey Project 

Questions to detect measurement error: 

□ Did the construction of the survey instrument follow a rigorous design procedure? Were cognitive 
interviews conducted? 

□ Is there any evidence that shows reliability in the survey data? 

□ Is there any evidence that shows validity in the survey data? 

□ Is there any pattern or tendency in occurrence of the responses? 

Questions to detect coverage error: 

□ Were the specifications of the target population clearly defined? 

□ Did the sampling frame include as many elements of the target population as possible? Did the 
parameters of the sampling frame correspond with the specifications of the target population? 

□ Were there any undercoverage cases, ineligible units or duplications in the sampling frame? 

Questions to detect sampling error: 

□ Was the sample size reasonable? (How many respondents did you expect to obtain? What margin of error 
was expected? What response rate was anticipated?) 

□ What sampling method was used? Was the method appropriate? Did you give equal chance of selection? 
For which subgroups did you give unequal chance of selection (if using stratification)? 

□ With the number of the respondents and the total target population, what was the margin of error? Was 
the obtained margin of error reasonable? 

Questions to detect non-response error: 

□ Was the response rate reasonable? 

□ Considering the topic of the survey, who may have been more likely to respond to the survey and who 
may have been less likely to respond? 

□ How did the profile of the respondents compare with that of the target population (or sampling frame) 
by certain demographic characteristics of interest? 

□ How were key subgroups represented in the respondents? 

□ Are there any flaws in questionnaire design and/or survey administration that may have resulted in non- 
responses by some survey recipients? 

□ Are there any question items that have a relatively large proportion of missing data? What may have 
been the causes? Did you report the missing data? 

Questions to detect post-survey error: 

□ How were the data cleaned? Was the procedure appropriate? 

□ How were the data coded? Was the procedure appropriate? 

□ How were the data weighted (if applicable)? Was the method correct? 

□ How were the data imputed (if applicable)? Was the method appropriate? 

□ What statistical techniques were used? Had the assumptions been inspected? Were the statistical 
procedures properly implemented? 
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